Determining Socio-Technical Systems Requirements: Experiences with Generating and Walking through Scenarios

Scenarios are effective for discovering requirements, but we still do not understand what types of scenario and which walkthrough techniques are most effective. This experience paper reports the application of one scenario approach – CREWS-SAVRE – to discovering requirements for naval and air traffic management systems with BAE SYSTEMS and Eurocontrol respectively. Results from these experiences are used to investigate important questions about the effectiveness of structured scenario walkthroughs and the level of domain-specificity most beneficial to the discovery of requirements. Lessons learned suggest that systematic walkthroughs of simple scenarios that do not contain excessive domain knowledge are more effective for discovering system requirements. The paper provides the reader with simple-to-use guidelines for scenariobased requirements discovery. 1. Discovering Requirements with Scenarios We know that scenarios are one of the more effective techniques for discovering stakeholder requirements [1]. However, scenarios can differ widely in their levels of abstraction and forms of representation, from narrative stories of how a system will behave [2] to computer-based animations of a system’s dynamic behaviour [3]. We can also use scenarios in different ways, from helping us to understand a current system [4] to walkthroughs of a future system’s behaviour to discover requirements for it [5]. However, there are no models and little reported evidence that tell us whether structured scenario walkthroughs and scenarios with more or less domain content lead to better requirements discovery. This paper reports results from recent, largescale case studies of scenario-driven requirements processes to provide some answers. The CREWS-SAVRE process and software tool were developed to deliver structured scenario walkthroughs with which to discover more complete stakeholder requirements [6, 7]. Design of its walkthrough tool was based on one cognitive principle often exploited during prototyping – that people recognise items, for example events, better than they recall them from memory [8]. CREWS-SAVRE generates and presents candidate alternative course events to stakeholders. The stakeholders then recognise which alternative courses are relevant to the new system and, where relevant, generate new requirements for them. This approach contrasts with existing use case methods (e.g. [2]) and scenario software tools (e.g. [9]) that primarily rely on people to recall the alternative courses themselves. Because people are better at recognising that recalling, we hypothesise that CREWS-SAVRE’s guided scenario walkthroughs can help stakeholders to discover more complete requirements. However, the lack of large-scale case studies has meant that we have not been able to test this hypothesis empirically. We also lack empirical data that indicates how domain-specific scenarios should be to discover requirements cost-effectively. For example, a simple scenario for an air traffic management (ATM) system might be expressed as an air traffic controller communicating with an aircraft and the communication failing. But the same scenario could also be expressed as a tactical controller working the North Sea Sector at London ATC issuing a direction change to 270 to flight BA123, and the congested frequency meaning that the instruction is not issued successfully. The second scenario probably needs more time and effort to produce, but does its more concrete content improve requirements discovery, and if it does, is the extra work needed to write it cost-effective? Case studies data are also needed to provide answers to these questions. This paper reports results from two applications in which CREWS-SAVRE was applied to acquire requirements for a new naval system and a new ATM system. In both applications we extended the CREWSSAVRE tool to generate domain-specific scenarios in the style of the North Sea sector scenario above. We then used the data about the resulting requirements and how they were arrived at to investigate whether structured scenario walkthroughs led to the discovery of more stakeholder requirements. We also used the data to investigate whether domain-specific scenarios enable stakeholders to discover more requirements, and whether generated CREWS-SAVRE scenarios can be specialised to the application domains at reasonable time and cost. The remainder of the paper is in 9 sections. The next two describe CREWS-SAVRE and the two application domains. Section 4 presents a simple predictive model of the scenario walkthrough process. Sections 5, 6 and 7 describe how we specialised CREWS-SAVRE to the domains and how effective this specialisation was for requirements discovery. Section 8 draws some conclusions and reports lessons to apply when using scenarios for discovering requirements. The paper ends by describing ongoing and future technical extensions of CREWS-SAVRE and more controlled empirical studies of its use. 2. Generating and Walking Through Scenarios with CREWS-SAVRE CREWS-SAVRE is a process with software tools that was developed as part of the EU-funded Framework IV 'CREWS' long-term research project to support systematic, domain-independent scenario generation and walkthrough process [7]. Features include: • Guidelines for writing use cases; • Automatic generation of scenarios from use cases; • Automatic generation of alternative courses; • Guided scenario walkthroughs. A project team writes use case descriptions using the structured templates, and style and content guidelines from the CREWS-ECRITOIRE method [10], enhanced with temporal action-ordering rules. This description is then parameterised and entered into the CREWSSAVRE tool, from which CREWS-SAVRE’s two-step scenario generation algorithm generates one or more scenarios. In the first step, the algorithm generates normal course scenarios from the action ordering rules and generation parameters in the use case specification. Each different possible ordering of normal course events is a different scenario. In the second step, the algorithm generates candidate alternative courses, which are expressed as ‘what-if’ questions for each normal course event, by querying a database that implements a simple model of abnormal behaviour and state in socio-technical systems. The model specifies 54 classes of abnormal behaviour and state using the structure shown in Figure 1. Some class hierarchies were derived from definitions of CREWS-SAVRE scenario concepts such as events and actions. Others were derived from error taxonomies in the cognitive science, human-computer interaction and safety-critical disciplines, and are reported at length in [6]. For example, the algorithm generates alternative courses about events not occurring and actions not completing by instantiating the ECand AC-coded classes in Figure 1, and about human agent mistakes, machine failures and interaction failures by instantiating the PE1-, PE2and PE3-coded sub-classes in Figure 1. Agents and objects Human-human communication Environment Events and actions Abnormal behaviour/state Mach-machine communication Humn-machine interaction Machine Human PE1 PE2 PE3 PE4 PE5 PE6 PE0